In-Camera Based Method of Detecting Defect Eye with High Accuracy

ABSTRACT

A portable digital image acquisition device includes multiple lenses and/or multiple flashes. A main digital image and first and second reference images are acquired. The first and second reference images are acquired with different flash-lens combinations that have different flash-lens distances. One or more flash eye defects are detected and corrected in the main image based on analysis of the first and second reference images.

PRIORITY

This application is a Division of U.S. patent application Ser. No.12/191,304, filed Aug. 14, 2008; which is incorporated by reference.

BACKGROUND OF THE INVENTION

It is known to capture non-flash images followed by flash images and bysubtracting these images (including some luminance normalizing), one canget a difference image which indicates where red-eye defects(flash-eyes) are located. U.S. Pat. Nos. 7,027,662 and 6,859,565 areincorporated by reference.

In addition, US published patent application 2005/0041121, entitled“Red-eye filter method and apparatus”, which is assigned to the sameassignee as the present application and incorporated by reference,describes techniques wherein a reference non-flash and the main flashimage may initially be of different sizes, and where the images aresized to the same dimensions and aligned before luminance normalizationand/or subtraction.

It is desired to have an improved technique involving capturing twoimages that are normalized for luminance and then subtracted, indicatingthe likely eye locations in a main acquired image, followed byacquisition of a main image. The process would be performed quickly(typically less than 1 second) to minimize any misalignment of thereference pair and the main acquired image.

It is also desired to have a technique that overcomes problemsassociated with the non-flash image tending to be significantly darkerthan the flash image and, particularly in indoor environments, whereinit is difficult to achieve practical luminance normalization.

SUMMARY OF THE INVENTION

A portable digital image acquisition device is provided. The deviceincludes a lens and a sensor for acquiring a main digital image andfirst and second reference images. The device also includes first andsecond light sources disposed at different distances from the lens forilluminating one or more objects within an acquired scene. The mainimage is stored in a memory. The device also includes one or moreprocessor-readable media having digital code embedded therein forprogramming a processor to perform a method of determining andcorrecting a flash eye defect in the main digital image using the firstand second reference images. The method includes acquiring the first andsecond reference images respectively using the first and second lightsources. The first and second images are normalized to generatenormalized first and second reference images. One or more differencesbetween the first and second normalized reference images are analyzed.The method further includes determining and correcting the flash eyedefect within the main image based on the analyzing to generate acorrected main image. The corrected main image or a further processedversion is stored, transmitted, communicated, displayed, and/orprojected.

Another portable digital image acquisition device is provided. Thisdevice includes first and second lenses and at least one sensor foracquiring a main digital image and first and second reference images.The device also includes a light source disposed at different distancesfrom the first and second lenses respectively for illuminating one ormore objects within an acquired scene. The main digital image and/or afurther processed version is stored in a memory. The device includes oneor more processor-readable media having digital code embedded thereinfor programming a processor to perform a method of determining andcorrecting a flash eye defect in the main digital image using the firstand second reference images. The method includes acquiring the first andsecond reference images using the light source and the first and secondlenses respectively. The first and second images are normalized togenerate normalized first and second reference images. One or moredifferences between the first and second normalized reference images areanalyzed. The method further includes determining and correcting theflash eye defect within the main image based on the analyzing togenerate a corrected main image. The corrected main image and/or furtherprocessed version is stored, transmitted, communicated, displayed,and/or projected.

In either device, the analyzing may include subtracting the first andsecond reference images.

The normalizing may include luminance normalizing.

The flash eye defect may include a red eye defect.

The normalizing may include re-sizing one or both of the first andsecond reference images.

The normalizing may include aligning the first and second referenceimages. The aligning may include aligning one or more eyes within thefirst and second images.

The differences may include color differences and/or brightnessdifferences.

The first and second reference images may be each acquired usingrelatively low-intensity pre-flashes compared with an intensity of aflash used in acquiring the main digital image.

The analyzing may take into account the specific different distancesbetween the first and second light sources and the lens in the firstdevice, or between the first and second lenses and the light source inthe second device.

The analyzing may further take into account distance to one or moreobjects, a gaze angle, an ambient lighting condition, color of an irisand/or a skin tone of a face within the acquired scene.

The first and second preview images may be acquired consecutively priorto acquiring the main digital image.

Methods determining and correcting a flash eye defect in the maindigital image using the first and second reference images are alsoprovided. Computer readable media having embedded code for programming aprocessor to perform the methods are also provided.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1A illustrates a camera with a flash close to an optical lens foracquiring an image.

FIG. 1B illustrates a camera with a flash farther from an optical lensthan the flash illustrated in FIG. 1A.

FIGS. 2A, 3A and 4A illustrate flash eye defects in digital imagesacquired with the camera of FIG. 1A.

FIGS. 2B, 3B and 4B illustrate flash eye defects in digital imagesacquired with the camera of FIG. 1B.

FIG. 5 illustrates a camera with two flashes separated at differentdistances from the camera's lens.

FIG. 6 illustrates a camera with two optical lenses separated atdifferent distances from the camera's flash.

FIG. 7 is a block diagram illustrating a method in accordance withcertain embodiments.

FIG. 8 is a block diagram illustrating a method in accordance withfurther embodiments.

DETAILED DESCRIPTION OF THE EMBODIMENTS

It is recognized herein that if a light source is located at a differentspatial distance from an imaging lens, the color and brightness of eyeregions in the image can vary significantly. At the same time, theremainder of the image will not vary as much in color and brightness.

FIGS. 1A and 1B illustrate two cameras each having a single flash and asingle lens. However, the flash-lens distance is greater for the cameraof FIG. 1B than for that of FIG. 1A. FIGS. 2A, 3A and 4A are picturestaken of at least the eyes of three people with the camera of FIG. 1A.FIGS. 2B, 2C and 2D are pictures taken of at least the eye of the samethree people with the camera of FIG. 1B. FIG. 2A clearly shows a lighterred than the eyes of the same person captured in FIG. 2B. The eyes alsoappear to be brighter n FIG. 2A than in FIG. 2B. FIG. 3A clearly shows alighter red than the eyes of the same person captured in FIG. 3B. Theeyes also appear to be brighter n FIG. 3A than in FIG. 3B. The right eyein FIG. 4A taken with the camera of FIG. 1A appears as a lighter redthan the right eye of the same person in FIG. 4B taken with the cameraif FIG. 1B. The left eye in FIG. 4A appears golden and very bright,while the left eye in FIG. 4B appears red and not nearly as bright asthe left eye of FIG. 4A. The differences between the eyes captured inFIGS. 2A, 3A and 4A and those of FIGS. 2B, 3B and 4B is the differentdistances between the flashes and the optical lenses of the respectivecameras used as illustrated at FIGS. 1A and 1B. Other features of thepartial faces shown in FIGS. 2A and 2B do not appear to differsignificantly, and same for FIGS. 3A-3B and 4A-4B.

In accordance with certain embodiments, FIG. 5 schematically illustratesa camera with two distinct light sources 2 a and 2 b at differentdistances from a main lens 4. Two reference images, which may bepreview, post-view and/or concurrent view images, are acquired using alow-intensity pre-flash from the two distinct flash units 2 a, 2 b. Thecamera illustrated schematically at FIG. 5 also includes a photosensor 6such as a CMOS sensor aligned with lens 4 for digitally-capturing imagesincluding the preview images and main images intended to be stored,transmitted, displayed, projected, communicated and/or furtherprocessed. The camera also includes a processor 8 and one or moredigital media 10 having program code stored therein. These media 10and/or other media may be used to store image data as well. The cameracan have other features such as one or more ports for connecting toanother device such as a printer, PC, display device, another camera,phone, etc., with a cable or wirelessly, and/or for plugging in a flashcard or other mountable device.

With the camera 1A illustrated schematically at FIG. 5, methods ofdetecting and corrected all kinds of defect eyes with high accuracy areprovided on a digital flash camera. Certain embodiments take intogreater account a red eye flash defect by concentrating on differencesin color between the two preview or other reference images, while otherembodiments take into greater account other defects such as golden eye,white eye, and/or zombie eye. The method is based on the effect ofreflection from the eyes of the flash light to the CCD and/or CMOS ofthe camera. Such is a function of few parameters. That is, a defect eyedistribution is a function of flash-lens positioning, distance to thesubject, gaze angle, ambient lighting conditions, color of the iris,race, among other factors. The method of certain embodiments exploitsvariations of the defect eye distribution with the distance between thelens and the main light source of the camera (flash lamp, focusing LED,or any other source light).

The distribution of defects changes with the distance between the flashlamp and the lens, as does the character of the defects. The frequencyof getting other than red defect eyes decreases with the increasing ofthe distance between the flash and lens. In addition, the intensity ofthe defect eyes decreases with the increasing of the distance betweenthe flash and lens. For example, a distance between the first flash andthe lens may be 2 cm, while the distance between the second flash andlens may be 5 cm. An optimal positioning of the two light sources 2 a, 2b would provide a maximum difference between the distribution of thedefect eyes on the two preview or other reference images, and a minimumdifference between the rest of the preview images.

The two flash light sources may be identical light sources 2 a, 2 b,e.g., flash lamps, or two focusing lamps, or LEDs, on a same camera 1Aat two different positions relative to the lens 4. The light sources maydiffer and the camera 1A would have software that would take thedifferences into account. However, the two identical light sources,except as to position relative to lens 4, provide two almost identicalpreview or other reference images. In this way, it is possible andoptimal to get the difference map of the two previews.

The camera 1A is able to take two consecutive preview pictures, justbefore the final flash picture, or two post-view pictures, or one ofeach. The first preview or other reference image is taken with the firstlight source 2 a of the camera 1A, and the second preview or otherreference image is taken with the second light source 2 b.

A difference map of the two preview or other reference images isgenerated. The differences could be determined in various ways, e.g., onthe red channel only for detecting red eye defects, starting from thecloser source light preview, or on the luminance channel only for whiteeye defects or golden eye defects.

A first source light 2 a is preferably located very close to the lens 4of camera 1A. For example, the light 2 a may be 1 cm or 2 cm on theright or left side of the lens as illustrated at FIG. 1 a. The secondlight source 2 b may be on the right or left side of the lens 4 a longerdistance such as 4 cm or 5 cm from the lens 4 such as illustrated atFIG. 2 b. A first preview image may be captured using flash 2 a, while asecond preview image may be captured using flash 2 b. Then the mainimage may be captured. As indicated, postview images may be capturedinstead. A difference map of the preview or postview images iscalculated. The difference map is filtered to eliminate smalldifferences, shadows, etc, and the significant differences that remainwill be analyzed as probably or at least possible defect eyes subject toflash eye defect correction algorithm. The sources 2 a, 2 b may be inthe simplest form two focusing lamps (e.g., LEDs) or in a complex formtwo flash lamps.

In accordance with further embodiments, FIG. 6 schematically illustratesa camera with a light source 12 and two distinct optical systems/lenses14 a and 14 b at different distances from the flash 12. Two previewimages may be acquired using a low-intensity pre-flash from the flashunit 12. The camera illustrated schematically at FIG. 6 also includes atleast one photosensor 6 aligned with lenses 14 a, 14 b, e.g., twoseparate photosensors one for each lens 14 a, 14 b or a single sensor,for digitally-capturing images including the preview images and mainimages intended to be stored, transmitted, displayed, projected,communicated and/or further processed. The camera also includes aprocessor 18 and one or more digital media 20 having program code storedtherein. These media 20 and/or other media may be used to store imagedata as well. The camera can have other features such as one or moreports for connecting to another device such as a printer, PC, displaydevice, another camera, phone, etc., with a cable or wirelessly, and/orfor plugging in a flash card or other mountable device.

In the embodiment of FIG. 6, a camera 1B has a single flash unit 12 andtwo imaging lenses 14 a, 14 b. A full dual imaging pipeline may beprovided as well. In this embodiment, a first preview image is acquiredusing one of the imaging lenses 14 a with a pre-flash illumination. Ator near the same time, a second preview image is acquired using thesecond imaging lens 14 b with illumination from the same weak pre-flash.The result is a pair of simultaneous preview images each being adifferent distance from the flash source. Any alignment errors will besolely due to geometrical factors rather than due to subject or cameramovement and a compensation algorithm can be pre-calibrated.

The reference images captured using the different flashes 2 a, 2 b ofthe camera of FIG. 5, or with the two sensors 14 a, 14 b with the cameraof FIG. 6, may then be compared, subtracted (with or without luminancenormalizing) and/or otherwise analyzed. As only the eye regions willhave very different characteristics, the locations of eye regions may bedetermined in this way. A third flash image may then be obtained using afull strength flash. This may be achieved by applying greater power toone of the flash units 2 a, 2 b, 12 or by combining the two flash units2 a, 2 b together in the camera of FIG. 5, or otherwise such as byincluding a third flash.

FIG. 7 illustrates a method involving the camera 1A of FIG. 5. First andsecond reference images are respectively acquired at 62 using the firstand second light sources and optical system 4 of camera 1A of FIG. 5.The first and second images are normalized to generate normalized firstand second reference images at 64. One or more differences between thefirst and second normalized reference images are analyzed at 66. A flasheye defect is determined and corrected within a main image based on theanalyzing to generate a corrected main image at 68. The corrected mainimage and/or a further processed version is/are stored, transmitted,communicated, displayed, and/or projected at 69.

FIG. 8 illustrates a method involving the camera 1B of FIG. 6. First andsecond reference images are acquired at 72 using a light source 12 andfirst and second lenses 14 a, 14 b of camera 1B of FIG. 6. First andsecond images are normalized at 74 to generate normalized first andsecond reference images. One or more differences between the first andsecond normalized reference images are analyzed at 76. A flash eyedefect within the main image is determined and corrected based on theanalysis to generate a corrected main image at 78. The corrected mainimage and/or a further processed version is/are stored, transmitted,communicated, displayed, and/or projected at 79.

While an exemplary drawings and specific embodiments of the presentinvention have been described and illustrated, it is to be understoodthat that the scope of the present invention is not to be limited to theparticular embodiments discussed. Thus, the embodiments shall beregarded as illustrative rather than restrictive, and it should beunderstood that variations may be made in those embodiments by workersskilled in the arts without departing from the scope of the presentinvention as set forth in the appended claims, and structural andfunctional equivalents thereof.

In addition, in methods that may be performed according to preferredembodiments herein and that may have been described above, theoperations have been described in selected typographical sequences.However, the sequences have been selected and so ordered fortypographical convenience and are not intended to imply any particularorder for performing the operations, except for those where a particularorder may be expressly set forth or where those of ordinary skill in theart may deem a particular order to be necessary.

In addition, all references cited above herein, as well as thebackground, invention summary, abstract and brief description of thedrawings, are all incorporated by reference into the detaileddescription of the preferred embodiments as disclosing alternativeembodiments. In addition, the following are incorporated by reference:

U.S. Pat. Nos. 6,407,777, 7,042,505, 7,352,394, 7,362,368, 7,269,292,7,369,712, 6,035,072, 7,336,821, 7,315,631, and 7,295,233,

United States published patent applications 2004/0223063, 2005/0041121,2005/0140801, 2006/0120599, 2006/0093213, 2007/0116379, 2006/0039690,2007/0116380, and 2007/0201724

U.S. patent applications Ser. Nos. 11/573,713, 11/462,035, 12/421,335,11/761,647, 11/753,098, 12/038,777, 12/043,025, 11/752,925, 11/836,773,11/767,412, 11/624,683, 60/945,558, 60/892,884, 11/861,257, 61/024,551,11/937,377, and 61/023,855.

1. A portable digital image acquisition device, comprising: (a) firstand second lenses and at least one sensor for acquiring a main digitalimage and first and second reference images; (b) a light source disposedat different distances from the first and second lenses for illuminatingone or more objects within an acquired scene; (c) a memory for storingthe main digital image; (d) one or more processor-readable media havingdigital code embedded therein for programming a processor to perform amethod of determining and correcting a flash eye defect in the maindigital image using the first and second reference images, wherein themethod comprises: (i) acquiring the first and second reference imagesusing the light source and the first and second lenses respectively;(ii) normalizing the first and second images to generate normalizedfirst and second reference images; (iii) analyzing one or moredifferences between the first and second normalized reference images;(iv) determining and correcting the flash eye defect within the mainimage based on the analyzing to generate a corrected main image; and (v)storing, transmitting, communicating, displaying, or projecting thecorrected main image or a further processed version, or combinationsthereof.
 2. The device of claim 1, wherein the analyzing comprisessubtracting the first and second reference images.
 3. The device ofclaim 1, wherein the normalizing comprises luminance normalizing.
 4. Thedevice of claim 1, wherein the flash eye defect comprises a red eyedefect.
 5. The device of claim 1, wherein the normalizing comprisesre-sizing one or both of the first and second reference images.
 6. Thedevice of claim 1, wherein the normalizing comprises aligning the firstand second reference images.
 7. The device of claim 6, wherein thealigning comprises aligning one or more eyes within the first and secondimages.
 8. The device of claim 1, wherein the differences comprise colordifferences.
 9. The device of claim 1, wherein the differences comprisebrightness differences.
 10. The device of claim 1, wherein the first andsecond reference images are each acquired using relatively low-intensitypre-flashes compared with an intensity of a flash used in acquiring themain digital image.
 11. The device of claim 1, wherein the analyzingtakes into account the specific different distances from the first andsecond lenses of the light source.
 12. The device of claim 11, whereinthe analyzing further takes into account distance to one or moreobjects, a gaze angle, an ambient lighting condition, color of an irisor a skin tone of a face within the acquired scene, or combinationsthereof.
 13. The device of claim 1, wherein the first and second previewimages are acquired consecutively prior to acquiring the main digitalimage.
 14. A method of determining and correcting a flash eye defect ina main digital image using first and second reference images, whereinthe method comprises: (i) acquiring first and second reference imagesusing a light source and first and second lenses respectively disposeddifferent distances from the light source; (ii) normalizing the firstand second images to generate normalized first and second referenceimages; (iii) analyzing one or more differences between the first andsecond normalized reference images; (iv) determining and correcting aflash eye defect within the main image based on the analyzing togenerate a corrected main image; and (v) storing, transmitting,communicating, displaying, or projecting the corrected main image or afurther processed version, or combinations thereof.
 15. The method ofclaim 14, wherein the analyzing comprises subtracting the first andsecond reference images.
 16. The method of claim 14, wherein thenormalizing comprises luminance normalizing.
 17. The method of claim 14,wherein the flash eye defect comprises a red eye defect.
 18. The methodof claim 14, wherein the normalizing comprises re-sizing one or both ofthe first and second reference images.
 19. The method of claim 14,wherein the normalizing comprises aligning the first and secondreference images.
 20. The method of claim 19, wherein the aligningcomprises aligning one or more eyes within the first and second images.21. The method of claim 14, wherein the differences comprise colordifferences.
 22. The method of claim 14, wherein the differencescomprise brightness differences.
 23. The method of claim 14, wherein thefirst and second reference images are each acquired using relativelylow-intensity pre-flashes compared with an intensity of a flash used inacquiring the main digital image.
 24. The method of claim 14, whereinthe analyzing takes into account the specific different distances fromthe first and second lenses of the light source.
 25. The method of claim24, wherein the analyzing further takes into account distance to one ormore objects, a gaze angle, an ambient lighting condition, color of aniris or a skin tone of a face within the acquired scene, or combinationsthereof.
 26. The method of claim 14, wherein the first and secondpreview images are acquired consecutively prior to acquiring the maindigital image.
 27. One or more processor-readable media having digitalcode embedded therein for programming a processor to perform a method ofdetermining and correcting a flash eye defect in a main digital imageusing first and second reference images, wherein the method comprises:(a) acquiring first and second reference images using a light source andfirst and second lenses respectively disposed at different distancesfrom the light source; (b) normalizing the first and second images togenerate normalized first and second reference images; (c) analyzing oneor more differences between the first and second normalized referenceimages; (d) determining and correcting a flash eye defect within themain image based on the analyzing to generate a corrected main image;and (e) storing, transmitting, communicating, displaying, or projectingthe corrected main image or a further processed version, or combinationsthereof.
 28. The one or more media of claim 27, wherein the analyzingcomprises subtracting the first and second reference images.
 29. The oneor more media of claim 27, wherein the normalizing comprises luminancenormalizing.
 30. The one or more media of claim 27, wherein the flasheye defect comprises a red eye defect.
 31. The one or more media ofclaim 27, wherein the normalizing comprises re-sizing one or both of thefirst and second reference images.
 32. The one or more media of claim27, wherein the normalizing comprises aligning the first and secondreference images.
 33. The one or more media of claim 32, wherein thealigning comprises aligning one or more eyes within the first and secondimages.
 34. The one or more media of claim 27, wherein the differencescomprise color differences.
 35. The one or more media of claim 27,wherein the differences comprise brightness differences.
 36. The one ormore media of claim 27, wherein the first and second reference imagesare each acquired using relatively low-intensity pre-flashes comparedwith an intensity of a flash used in acquiring the main digital image.37. The one or more media of claim 27, wherein the analyzing takes intoaccount the specific different distances from the first and secondlenses of the light source.
 38. The one or more media of claim 37,wherein the analyzing further takes into account distance to one or moreobjects, a gaze angle, an ambient lighting condition, color of an irisor a skin tone of a face within the acquired scene, or combinationsthereof.
 39. The one or more media of claim 27, wherein the first andsecond preview images are acquired consecutively prior to acquiring themain digital image.